JOURNAL OF LEARNING ANALYTICS 


S8LAR 

SOCIETY for LEARNING 
ANALYTICS RESEARCH 


(2015). Revisiting for retention: An analytic for inquiry science learning. Journal of Learning Analytics, 2(2), 75-101. 

http://dx.doi.Org/10.18608/jla.2015.22.7 

Distributed Revisiting: An Analytic for Retention of Coherent 

Science Learning 

Vanessa Svihla and Michael J. Wester 

University of New Mexico, Albuquerque, USA 
vsvihla@unm.edu 

Marcia C. Linn 

University of California, Berkeley, USA 


ABSTRACT: Designing learning experiences that support the development of coherent 
understanding of complex scientific phenomena is challenging. We sought to identify analytics 
that can also guide such designs to support retention of coherent understanding. Based on prior 
research that distributing study of material over time supports retention, we explored revisiting 
previously studied material as an analytic. We tested ways to operationalize revisiting: as a 
general propensity to revisit previously studied material; as a propensity to revisit specific 
curricular steps; as a general propensity to distribute study by revisiting previously studied 
material on different days; and as a propensity to distribute study by revisiting specific steps on 
different days. The specific steps identified as central to the learning design included a static 
illustration and a dynamic visualization. We modelled revisiting in a sample of 664 students 
taught by seven different teachers using a Web-based Inquiry Science Environment unit. Analysis 
of log files and regression modelling revealed that a general propensity to revisit did not predict 
retention. Revisiting the dynamic visualization better supported retention than revisiting static 
material, but only for distributed revisiting. Our findings suggest that revisiting can be a useful 
analytic when aligned with the framework guiding learning design. 

Keywords: Learning design, revisiting, distributed practice, knowledge integration, inquiry 
science 

Editor's Note: As part of the Special Section on Learning Analytics & Learning Theory this article is followed by a 
short commentary on pp. 102-106 that discusses the challenges it faced and successes it achieved in drawing on 
and contributing to theory use in learning analytics. 

1 RATIONALE 

Lockyer, Heathcote, and Dawson (2013) suggest a symbiotic relationship between learning design and 
learning analytics, with the former providing a lens to understand pedagogical aims and the latter as 
evidence that can be used to evaluate the design; this approach is compelling because data are collected 
passively and in-situ. The learning design necessarily informs the pedagogical aims, as without this, 
deciphering the relevance of particular data signals is challenging. As a result, much of learning analytics 
has focused on student progress and time spent on a task. 
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There is evidence that spending more time on task can support learning and retention (Barbera & 
Reimann, 2013; Cotton, 1990). Using amount of time as a metric for learning is appealing because it is 
relatively easy to measure. However, what students do during time spent studying also matters. To 
develop a more nuanced metric, we consider findings from studies of distributed practice , one of the 
most well-documented findings from laboratory studies of recall; numerous studies show that restudy 
of material, distributed in time, supports learning (Cepeda, Pashler, Vul, Wixted, & Rohrer, 2006; 
Delaney, Verkoeijen, & Spirgel, 2010; Donovan & Radosevich, 1999; Janiszewski, Noel, & Sawyer, 2003). 
In this study, we explore the metric of revisiting — how and what students spontaneously revisit in 
inquiry science materials — and how it relates to their retention. We specifically explore ways revisiting 
can be a useful analytic for guiding the design of technology-enhanced learning experiences when 
retention of coherent understanding of complex scientific phenomena is the learning goal. This responds 
to calls for learning analytics to guide learning design (Ferguson, 2012). 

2 LITERATURE REVIEW 


We begin by discussing the Knowledge Integration framework that guided our learning design (Figure 1). 
We review literature on distributed practice to explain why we sought revisiting as a relevant metric for 
retention. We then discuss research on static and dynamic visualizations for learning inquiry science, as 
we compare these as targets for students' spontaneous revisiting in web-based learning environments. 


Knowledge Integration framework 

Build initial coherent understanding 


Elicit_(.Add _(.Distinguish_> Link 

initial new ideas ideas 

ideas ideas 

Pattern repeats for each activity in unit j 


Develop coherent understanding of 
complex phenomena 

Revisit and link ideas across activities, 
distributed over time 



-► 

Time on task 

Students need time to link complex ideas 


Figure 1. Model for retention of coherent understanding of complex phenomena, informed by the 
Knowledge Integration framework and research on distributed practice. The left-hand figure 
represents a single activity within a unit. In the right-hand figure, the arrows indicate that students 
can directly revisit different steps within an activity. 


2.1 Knowledge Integration Framework Guiding Learning Design 


We use the term learning design (Laurillard, 2012; Lockyer et al., 2013) to describe the pedagogical 
approach — Knowledge Integration — instantiated in a Web-based Inquiry Science Environment (WISE) 
(Slotta & Linn, 2009) unit. Learning designs aim to be reusable or adaptable across contexts, and WISE 


ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 


76 











JOURNAL OF LEARNING ANALYTICS 


S8LAR 


(2015). Revisiting for retention: An analytic for inquiry science learning. Journal of Learning Analytics, 2(2), 75-101. 

http://dx.doi.Org/10.18608/jla.2015.22.7 


units accomplish this by providing professional development for teachers and providing tools that allow 
teachers to adapt units for their classrooms; this adaptation commonly continues into the classroom as 
teachers implement units differently, with some teachers weaving other resources, quizzes, labs, and 
activities into the unit, and others taking a guiding role, offering additional instruction only when 
students seek help. WISE units provide guidance to teachers and students about what they should do 
and when to do it, and the resources required to carry out the activities, which are primarily simulation- 
based. 


The Knowledge Integration framework (Kali, 2006; Linn & Eylon, 2011) draws on extensive classroom 
research to identify ways to guide students to integrate their diverse and often conflicting ideas about 
core themes, such as energy transfer and transformation across science disciplines (Svihla et al., 2010). 
We refer to the process by which students develop a coherent understanding of science that involves 
generating ideas, adding new ideas, comparing and distinguishing their new and prior ideas, and linking 
relevant, normative ideas together, as knowledge integration (Kali, Linn, & Roseman, 2008). 

WISE units scaffold students using an inquiry map (Figure 2) to support them in developing coherent 
understanding of science ideas (Linn, 2006). Each unit comprises activities of multiple steps (listed along 
the left side). Activities first elicit students' ideas. Next, activities introduce students to new ideas 
(commonly described in the Knowledge Integration framework as "adding ideas"). Finally, activities help 
students distinguish and evaluate between their initial ideas and newly added ideas. In many activities, 
this is accomplished through sequences in which students make predictions, interact with dynamic 
visualizations, then reflect on their observations. For instance, in the unit called Global Climate Change 
(GCC) (Svihla & Linn, 2012a), students learn about the greenhouse effect and the role of energy transfer 
and energy transformation in climate change. They investigate NetLogo visualizations (Wilensky & 
Reisman, 2006) representing the earth and atmosphere, and variables involved in climate change. 



Figure 2. Inquiry map for the WISE 4 environment. Activities five through seven are visible; activity 
seven is expanded to show it comprises seven steps. 
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Using the inquiry map, students can revisit prior steps throughout the unit; this is part of the learning 
design, as one goal of the knowledge integration framework is to promote the ability of students to 
monitor their own learning by knowing when they need more information (Chiu, 2010). More generally, 
recognizing the value of revisiting is consistent with recognizing the need to stop and explain a text to 
oneself, a process called "generating self-explanations" (Chi, Bassok, Lewis, Reimann, & Glaser, 1989). 
Some research suggests that students are able to judge when it is advantageous to distribute their 
learning over time (Popham, 2009). However, studies of self-explanation show that some students 
spontaneously seek to explain conundrums while others do not (Slotta & Chi, 2006) and that individuals 
differ in their propensity to update their memories (Bjork, 1978). For this reason, some authors of WISE 
units structure revisiting, such as by forcing students to revisit earlier steps based on non-normative 
responses. However, understanding students' spontaneous revisiting patterns and their relationship to 
retention of coherent understanding could help guide learning designs beyond simple checks. For 
instance, prior research has demonstrated that distributing study over time better supports retention; 
thus, understanding how students revisit steps over time could guide new design patterns that better 
support retention. 

2.2 Revisiting as Distributed Practice to Support Retention 

2.2.1 Revisiting as Distributed Practice Supports Learning 

Students might revisit a step to help clarify their ideas, making revisiting a form of distinguishing ideas, 
as advocated in the knowledge integration framework (Svihla & Linn, 2012b; Zhang & Linn, 2011). 
However, revisiting could also be unproductive, as when students reread text, adding multi-coloured 
underlining but not gaining more insight into the material (Bjork & Bjork, 2009). There are many reasons 
why a student might decide to revisit a step; yet, any revisiting provides an opportunity to restudy. Thus, 
a general propensity to revisit previously studied material could confer an advantage for retention. 

Students develop more durable, integrated understanding when they distribute their study in time, 
rather than when they mass their practice. For instance, consider two students studying for a chemistry 
test. Diego reviews the material for 10 minutes each day, for seven days. Mario reviews the material for 
70 minutes the day before the test. Hundreds of studies (e.g., as reviewed in Cepeda et al., 2006; 
Delaney et al., 2010; Donovan & Radosevich, 1999; Janiszewski et al., 2003) would predict Diego will do 
better on the exam because he distributed his study over time. These findings held for repetition and 
induction tasks (Kornell, Castel, Eich, & Bjork, 2010) and abstraction and generalization tasks (West, 
2011). Classroom studies and studies with educationally relevant materials have recently become more 
common, finding benefits to distributed study of scientific prose (Roediger & Karpicke, 2006), maps 
(Carpenter & Pashler, 2007), history facts (Carpenter, Pashler, & Cepeda, 2009), vocabulary (Bloom & 
Shuell, 1981; Seabrook, Brown, & Solity, 2005; Sobel, Cepeda, & Kapler, 2010), multiplication facts (Rea 
& Modigliani, 1985), statistics concepts (Bude, Imbos, Wiel, & Berger, 2011; Smith & Rothkopf, 1984), 
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middle school biology concepts (Reynolds & Glaser, 1964), university medical education (Kerfoot, 
Kearney, Connelly, & Ritchey, 2009), and elementary science (Vlach & Sandhofer, 2012). 

However, many of these studies occurred over a relatively short period, raising the need for further 
research under real-world and longer timeframe conditions (Cepeda et al., 2006; Dunlosky & Rawson, 
2012). Partially taking up these concerns, recent research has documented the utility and generality of 
learning analytics approaches to distributed practice, finding, for instance, that learning management 
systems can provide usable information about distributed practice (Andergassen, Modritscher, & 
Neumann, 2014; Modritscher, Andergassen, & Neumann, 2013). These studies show that distributing 
study across time and over days supports learning for a range of topics, particularly when students are 
able to integrate ideas across a course of study (Andergassen et al., 2014). However, questions still 
remain about how to support students to revisit previously studied content effectively to lead to 
retention of integrated understanding, which may depend on more than simple revisiting to recall 
previously studied material (Dunlosky & Rawson, 2012). 

2.3 Retention of Integrated Understanding 

Researchers have varied the length of time between the final study session and the delayed post-test — 
called the retention interval (Carpenter, Cepeda, Rohrer, Kang, & Pashler, 2012). Several studies of 
retention have produced reversal effects from immediate post-test to delayed post-test (Bird, 2010; 
Rawson, 2012; Rawson & Kintsch, 2005). In these studies, distributed study sessions do not appear to 
benefit performance on the immediate post-test, but they do on delayed post-tests. For instance, in a 
study involving calculating the number of permutations for sequences, students first learned how to do 
a problem, and then either completed ten practice problems in one clustered session or in two sessions 
one week apart. Post-test results showed no difference, but the delayed post-test given four weeks later 
showed greater retention for those in the distributed group (Rohrer & Taylor, 2006). 

Likewise, in a study of expository texts contrasting a massed condition with a distributed condition with 
a one-week gap between activities, an advantage was found after a longer retention interval. Students 
were tested for reading comprehension either immediately upon completion or after a two-day 
retention interval (Rawson & Kintsch, 2005). Those in the massed condition performed better on the 
immediate test, but those in the distributed condition performed better after a two-day retention 
interval. To explain why this might occur, researchers explored whether the long or short retention 
interval would lead to more integrated ideas (Rawson, 2012). Rawson tested this idea by focusing on 
specific post-test question types (free and cued recall), predicting that longer gaps between study 
sessions and a longer retention interval would depend "more heavily on the degree of integration" 
(Rawson, 2012, p. 873). Analysis showed that participants with longer gaps between study sessions were 
likelier to recall the main and important ideas over the unimportant ideas. The students experiencing a 
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longer retention interval performed significantly higher on the cued recall questions by recalling more 
ideas after the delay. 

Thus, the benefits of revisiting previously studied material might not show up on immediate post-tests. 
A delayed post-test is likelier to detect the potential benefit of distributed study. This presents a 
potential challenge to the design of adaptive learning systems, unless retention is a priority of the 
designers. 

2.4 Revisiting What? 

While we cannot always detect why students choose to revisit previously studied material, we can easily 
explore what they revisit, and test revisiting as a diagnostic for retention of coherent understanding. 
WISE units include various curricular step types, such as text, static illustrations, and dynamic 
visualizations. 

Based on prior research, we know that students can learn about abstract or invisible phenomena — like 
heat transfer and transformation — from dynamic visualizations (e.g., Cook, 2006; Marbach-Ad, Rotbain, 
& Stavy, 2008). While there is continued controversy about how and when static versus dynamic 
visualizations support learning (Tversky, Morrison, & Betrancourt, 2002), there is substantial support for 
their use in learning (Hoffler & Leutner, 2007), particularly when student interactions with visualizations 
are scaffolded (Hegarty, 2004; Tversky et al., 2002). In our prior research, we have found that students 
do learn from dynamic visualizations designed to support knowledge integration (Ryoo & Linn, 2010, 
2012; Svihla & Linn, 2012a). This typically means that students are first asked to make predictions, 
interact with a dynamic visualization, and then reflect on what they learned. 

When supporting students to learn about a complex phenomenon, we have also added static 
illustrations to help students connect their prior experiences to the phenomenon and notice salient 
features when they interact with the dynamic visualizations. For instance, in the Global Climate Change 
unit, we added a static illustration (Figure 3, left) prior to the dynamic visualization (Figure 3, right). This 
was added based on analysis of video data and log file data from initial testing of the learning design. 

This combination supported the initial development of coherent understanding of climate change and 
energy transformation as an important mechanism for the greenhouse effect (Svihla & Linn, 2012a); 
here we explore how students' spontaneous revisiting of these relates to retention of this 
understanding. 
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Some scientists have found evidence for a cold climate followed by volcanic eruptions. They think 
volcanoes may have helped warm up the climate in the past. 

When a volcano erupts, greenhouse gases, such as C0 2 are released. In the model below, these are small 
black circles. 



Figure 3. On left, the static curriculum step in which students see a simple metaphor for energy 
transformation as an advance organizer. On the right, a dynamic visualization step, in which students 
first see a screenshot suggesting various experiments and then can test out their own ideas with the 

visualization. 


The static illustration contains information about energy transformation in the greenhouse effect and 
compares it to phase changes in water. The dynamic visualization contains information about energy 
transformation in the greenhouse effect, and additionally allows students to relate this to global 
temperature. In contrast to the static illustration, the information is not presented directly; students 
must explore the visualization to uncover this information. Thus, the dynamic visualization contains 
more information than the static illustration, but the information is available only through interaction 
with the dynamic visualization. 

3 RESEARCH PURPOSE 


We investigated students' spontaneous revisiting of prior steps as a form of self-distributed learning. We 
tested various types of revisiting of earlier steps in the unit with a large sample of students taught by 
several teachers. We also examined duration of instruction. We investigated the following questions: 

1. How does duration of instruction explain variance in retention of integrated knowledge? 

2. How do duration of instruction and specific ways to operationalize revisiting as an analytic explain 
variance in retention of integrated knowledge, with revisiting defined as propensity to: 

a) revisit prior steps in general; 
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b) revisit specific steps — a static illustration or a dynamic visualization; 

c) distribute study in general by revisiting steps across days; 

d) distribute study by revisiting specific steps — a static illustration or a dynamic visualization — 
across days. 

We hypothesized that propensity to distribute study by revisiting specific steps would best predict 
retention. We further predicted that restudy of the more complex dynamic visualization would better 
support retention. Operationalizing revisiting in this way best aligns to our learning design. 

4 METHODS 

4.1 Instructional Materials 

Students studied a previously tested WISE unit called Global Climate Change (GCC) (Svihla & Linn, 
2012a). This unit teaches students about the greenhouse effect and the role of energy transfer and 
energy transformation in climate change. 

4.2 Participants 

Participants were 835 grade 6 students (664 students completed all measures) taught by seven different 
teachers at three culturally diverse middle schools. All teachers taught the unit in five to seven 
consecutive class periods. 

Data were collected during two consecutive school years. Each teacher was assigned an ID (Table 1). 
Three teachers provided data for both years. Three of the teachers taught multiple class periods per 
day, resulting in larger sample sizes for those teachers. We define an implementation as including all 
class periods taught by a teacher in a given semester. In total, ten implementations are included in the 
study, with three teachers contributing two implementations. Mean total time spent studying the unit 
ranged from 133 to 301 minutes. 

4.3 Assessments 

We used Knowledge Integration assessments aligned with the unit. The assessment items have been 
shown to have good psychometric properties and to be valid indicators of knowledge integration (Liu, 
Lee, Hofstetter, & Linn, 2008; Liu, Ryoo, Linn, Sato, & Svihla, 2015). Retention was measured using a 
delayed post-test (M= 12.88, SD= 2.06) administered a mean of 23 days following instruction (Table 1). 
Teachers gave the delayed post-test to fit their particular schedules, resulting in retention intervals that 
varied from 4 to 40 days. The delayed post-test included six Knowledge Integration items (maximum 
possible score of 21) drawn from a longer assessment that additionally covered other topics as part of a 
project investigating cumulative learning (Liu et al., 2015; Svihla et al., 2010). The items included in the 
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delayed post-test assessed students' understanding of energy transfer by radiation, energy 
transformation, and the role energy transformation plays in the greenhouse effect; these same concepts 
are taught in the curriculum steps in Figure 3. The delayed post-test scores appear relatively low in part 
because energy transfer and transformation were challenging items, intended to detect growth over 
multiple years. 

Table 1. Descriptive statistics for each implementation, including retention interval, delayed post-test 

scores, and total minutes spent on the unit 


Year 

Teacher ID 

Retention 

interval 

n 

Delayed Post-test 

score 

Total minutes 



(days) 


M 

SD 

M 

SD 

2010 

0 

31 

79 

14.44 

2.64 

215.73 

24.67 

2010 

1 

17 

28 

15.00 

2.80 

229.69 

36.38 

2010 

2 

15 

60 

13.45 

2.59 

187.23 

48.07 

2010 

3 

23 

27 

13.89 

3.07 

184.19 

21.11 

2011 

0 

4 

93 

12.19 

1.66 

205.06 

45.66 

2011 

1 

12 

52 

13.02 

1.59 

300.75 

68.94 

2011 

2 

26 

91 

12.36 

1.42 

235.73 

59.19 

2011 

4 

24 

26 

12.31 

1.19 

222.47 

49.43 

2011 

5 

15 

26 

12.04 

1.08 

95.78 

17.90 

2011 

6 

40 

182 

12.36 

1.26 

132.62 

25.24 


5 ANALYSIS 

5.1 Data Coding 

Student responses were saved by the WISE4 system and scored using validated Knowledge Integration 
rubrics (Svihla & Linn, 2012a). Two raters coded a subset of items; any discrepancies were discussed 
until consensus was reached. To be successful on Knowledge Integration items, students must exhibit a 
coherent, connected, normative understanding (Table 2). Note that because the lowest possible score 
per question is 1, tests containing multiple questions have a minimum score equal to the number of 
items (as opposed to 0). 
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Table 2. Knowledge Integration scores, levels, and descriptions 


Knowledge 

Integration 

score 

Level 

Description 

1 

Irrelevant 

Does not answer the question being asked, or chose not to 

answer 

2 

Non-normative 

Contains non-normative ideas or links, vague ideas, or 
scientifically invalid connections between ideas 

3 

Partial link 

Unelaborated connections using relevant features OR 
Scientifically valid connections not sufficient to solve the 
problem. 

4 

Full link 

One scientifically complete and valid connection 

5 

Complex link 

Two or more scientifically complete and valid connections 


5.2 Log File Analysis 

The WISE system logs student progress through units, providing timestamps of the students' activity in 
sequence in an exportable Excel file (Figure 4). All student dyads associated with a teacher's 
implementation are exported into a single file. To reliably document student revisiting, we used an 
AppleScript (excel2csv.app) to extract student logs into individual .csv files. We then used MATLAB to 
extract data about revisiting patterns and timing. This process resulted in a .csv file for each teacher's 
implementation. 


# 

Step Title Step Type 

Step Prompt 

Start Time 

End Time 

Time 

Student Work 

1 

1.1 Introduction Html 

N/A 

2010-05-06 16:27:06.0 

2010-05-06 16:27:41.0 

35 

N/A 

2 

1.2 Pre: Car on col Assessment At the end of the 2010-05-06 16:27:41.0 

2010-05-06 16:30:22.0 

161 

Warmer than t 

3 

1.3 Pre: Burning C Assessment 

Note: At the end (2010-05-06 16:30:22.0 

2010-05-06 16:34:36.0 

254 

Lower relative 

4 

1.4 Pre: Advise Gw Assessment 

<p>Note: At the e 2010-05-06 16:34:36.0 

2010-05-06 16:38:54.0 

258 

Turn off com pi 

5 

1.5 MySystem Inst Html 

N/A 

2010-05-06 16:38:54.0 

2010-05-06 16:39:26.0 

32 

N/A 

6 

1.6 Pre: MySysterr MySystem 

2010-05-06 16:39:26.0 

2010-05-06 16:39:44.0 

18 

N/A 

7 

1.5 MySystem Inst Html 

N/A 

2010-05-06 16:39:44.0 

2010-05-06 16:39:54.0 

10 

N/A 

8 

1.6 Pre: MySysterr MySystem 

2010-05-06 16:39:54.0 

2010-05-06 16:42:36.0 

162 

N/A 

9 

1.5 MySystem Inst Html 

N/A 

2010-05-06 16:42:36.0 

2010-05-06 16:42:44.0 

8 

N/A 

10 

1.6 Pre: MySysterr MySystem 

2010-05-06 16:42:44.0 

2010-05-06 16:43:14.0 

30 

N/A 

11 

1.5 MySystem Inst Html 

N/A 

2010-05-06 16:43:14.0 

2010-05-06 16:43:19.0 

5 

N/A 

12 

1.6 Pre: MySysterr MySystem 

2010-05-06 16:43:19.0 

2010-05-06 16:43:21.0 

2 

N/A 

13 

1.5 MySystem Inst Html 

N/A 

2010-05-06 16:43:21.0 

2010-05-06 16:43:24.0 

3 

N/A 

14 

1.6 Pre: MySysterr MySystem 

2010-05-06 16:43:24.0 

2010-05-06 16:44:43.0 

79 

N/A 

15 

1.5 MySystem Inst Html 

N/A 

2010-05-06 16:44:43.0 

2010-05-06 16:44:47.0 

4 

N/A 

16 

1.6 Pre: MySysterr MySystem 

2010-05-06 16:44:47.0 

2010-05-06 16:45:03.0 

16 

N/A 

17 

1.5 MySystem Inst Html 

N/A 

2010-05-06 16:45:03.0 

2010-05-06 16:45:07.0 

4 

N/A 

18 

1.6 Pre: MySysterr MySystem 

2010-05-06 16:45:07.0 

2010-05-06 16:45:54.0 

47 

N/A 

19 

1.6 Pre: MySystem MySystem 

2010-05-07 12:45:51.0 

2010-05-07 12:46:55.0 

64 

N/A 

20 

1.5 MySystem Inst Html 

N/A 

2010-05-07 12:46:55.0 

2010-05-07 12:47:03.0 

8 

N/A 

21 

1.6 Pre: MySystem MySystem 

2010-05-07 12:47:03.0 

2010-05-07 12:47:16.0 

13 

N/A 

22 

1.5 MySystem Inst Html 

N/A 

2010-05-07 12:47:16.0 

2010-05-07 12:47:20.0 

4 

N/A 


Figure 4. Screenshot of an Excel workbook containing log file data from one teacher's implementation 

of the Global Climate Change unit. 


The first column in the spreadsheet provides the sequence (Figure 4). The second column identifies the 
number and title of the step the dyad visited. The third and fourth columns provide information about 
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the step type and prompts. The fifth, sixth, and seventh columns provide time and date information 
about each visit. The eighth column shows student work, making it possible for researchers to identify 
when students made changes to their answers. 

Thus, several variables are easily computed. We computed the total time (duration of study) each dyad 
spent on the unit. To explore revisiting, we computed several variables (Tables 3 and 4). We first 
eliminated visits lasting less than five seconds since these generally resulted from students clicking the 
"Next" or "Back" buttons in rapid succession. Rapid clicking was also observed during classroom 
observations and in videos of implementation when students quickly clicked through steps to reach a 
desired step. 

Based on past research showing that students differ in their propensity to update their memories (Bjork, 
1978), we computed a revisiting disposition variable, defined as the average number of visits made to 
steps across the unit (Tables 3 and 4). Thus, if a dyad visited every step in the unit twice, they would 
receive a score of two. 

We also hypothesized that while students differ in propensity to update their memories, they could also 
differ in what they chose to revisit. We viewed the specific curriculum steps in Figure 3 as likely targets 
of restudy based on classroom observations. These two steps in particular are the central focus of the 
unit. They also provide an interesting contrast. Previous studies of dynamic visualizations embedded in 
WISE units have provided evidence that students often learn more from them compared to static 
illustrations. We anticipated differences when dyads revisited one or the other because of differences in 
the type of step (dynamic versus static) and the amount of information in each step (both contained 
information about the role of energy transformation in the greenhouse effect, but only the dynamic 
visualization related that to changes in global temperature). We calculated variables for the total 
number of visits to each of these steps. 

Based on past research, we also wanted to test the benefit for longer lags between revisits. For instance, 
based on a meta-analysis, longer intervals between study sessions better support more complex tasks 
(Donovan & Radosevich, 1999). This suggests that immediate revisits (e.g., those occurring on the same 
day) to recently viewed materials may not lead to longer retention of coherent understanding. Thus, we 
computed a distributed disposition variable based on the number of days each dyad visited each step. 
This was calculated as the average number of days students visited each step throughout the unit. 
Students who visited each step on one and only one day would have a score of one, whereas those who 
visited each step on two days would have a score of two. 

We also computed distributed visit variables for the same specific steps — the dynamic visualization and 
the static illustration. We predicted an advantage when students visited these steps on more than one 
day because the visits would be more distributed in time. 
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Table 3. Variables computed to explore revisiting 


Type of revisiting 

Calculated as 

Justification 

Revisiting disposition: 

General disposition to 

revisit 

Calculated as average number of 
visits across all steps. Both same- 
day and across-day visits to a step 

counted. 

Students differ in their propensity 
to update their memories (Bjork, 
1978). 

Number of visits to 

static illustration. 

Calculated as the total number of 

visits to the static curriculum step. 

Students vary in propensity to 
update their memories (Bjork, 
1978), but what they choose to 
revisit could correspond to what 
they retain. 

Number of visits to 

dynamic visualization. 

Calculated as the total number of 

visits to the dynamic visualization. 

Distributing disposition: 

Disposition to revisit 
steps across days 

Calculated as average number of 
days a dyad visited each step. 

Students can judge when it is 
advantageous to distribute their 
learning over time (Popham, 
2009). 

Number of days visited 

static illustration 

Calculated as the total number of 

days a dyad visited the static 

illustration. 

Students can judge when it is 
advantageous to distribute their 
learning over time (Popham, 
2009), but what they choose to 
revisit could correspond to what 
they retain. 

Number of days visited 
dynamic visualization 

Calculated as the total number of 

days a dyad visited the dynamic 

visualization. 


Table 4. Descriptive statistics for each type of revisiting variable 


Year 

Teacher ID 

Revisiting 

disposition 

Visits to Visits to 

static dynamic 

illustration visualization 

Distributing 

disposition 

Days visited Days visited 
static dynamic 

illustration visualization 

M 

SD 

M 

SD 

M 

SD 

M 

SD 

M 

SD 

M 

SD 

2010 

0 

1.97 

0.57 

1.31 

0.98 

3.73 

2.86 

1.18 

0.33 

1.18 

0.77 

1.94 

1.28 

2010 

1 

1.76 

0.24 

1.25 

0.51 

3.64 

1.98 

1.10 

0.11 

1.05 

0.23 

1.46 

0.5 

2010 

2 

1.8 

0.56 

1.32 

0.83 

3.08 

2.15 

1.11 

0.32 

1.11 

0.52 

1.61 

0.8 

2010 

3 

1.78 

0.4 

1.40 

0.62 

3.40 

2.77 

1.16 

0.11 

1.07 

0.25 

1.2 

0.41 

2011 

0 

2.15 

0.56 

1.21 

0.49 

3.04 

2.67 

1.00 

0.14 

1.06 

0.31 

1.32 

0.77 

2011 

1 

2.54 

0.59 

5.95 

2.67 

9.27 

4.15 

1.40 

0.22 

2.49 

0.74 

2.53 

0.79 

2011 

2 

1.56 

0.44 

1.60 

1.01 

2.88 

2.09 

0.89 

0.42 

1.25 

0.59 

1.69 

0.92 

2011 

4 

2.29 

0.74 

3.36 

2.23 

6.5 

4.25 

1.39 

0.38 

2 

0.86 

2.21 

0.96 

2011 

5 

1.56 

0.21 

3.90 

1.82 

5.76 

3.26 

0.97 

0.10 

2 

0.38 

2 

0 

2011 

6 

1.46 

0.3 

1.15 

0.66 

2.28 

1.86 

0.98 

0.15 

1.01 

0.31 

1.07 

0.39 
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5.3 Regression Modelling of Revisiting Practices 

We initially modelled delayed post-test scores using Hierarchical Linear Modelling (HLM), but the 
number of participants was too low due to missing data. Calculations of intra-class correlation indicated 
that less than 5% of the variance in delayed scores was explained by clustering. Other research has 
suggested that in such cases, multi-level modelling is not warranted (Lee, 2000). Based on these 
findings, we proceeded with ordinary least squares regression modelling. 

We know from prior work that students need sufficient time to develop coherent understanding of 
complex phenomena like climate change. We therefore first modelled retention using time spent on the 
unit. We tested various contextual variables (e.g., gender, teacher, school) not detailed here. We found 
that they did not explain significant variance. 

We proceeded to add, then remove, the revisiting variables stepwise, resulting in four models that 
tested revisiting: 

• Model 2: Revisiting disposition; 

• Model 3: Revisiting the static illustration and dynamic visualization; 

• Model 4: Distributing disposition; 

• Model 5: Distributed revisiting of the static illustration and dynamic visualization. 

Adding all variables at once would have resulted in multicollinearity, as the step-specific variables would 
not have explained variability not already explained by the disposition variables. Collinearity tolerances 
ranged from .56 to .85, suggesting that although the number of total visits to the two specific steps 
(r=.59) and number of days the specific steps were visited (r=.57) were significantly correlated to each 
other, this was not an issue in the stepwise approach. 

6 RESULTS OF REGRESSION MODELLING 

A simple linear regression was calculated to predict delayed post-test scores based on duration of 
instruction (M= 198 minutes, SD =65 minutes). The delayed post-test score had a mean of 12.88 across 
classes and a standard deviation of 2.06. A significant regression equation was found (F(l, 662)= 12.99, 
p<.001) (Model 1, Table 5, Figure 5). Spending more time on the unit in total predicted higher scores on 
the delayed post-test, with an increase in the delayed post-test score of .004 for each additional minute 
spent. This was statistically significant but accounted for a small amount of variance in delayed post-test 
scores, r 2 = .02, p<.05. 
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Total minutes 

Figure 5. Scatterplot showing the relationship between total time spent on the unit and scores on the 

delayed post-test. Trend line shows r 2 = .19. 

In Model 2, a multiple linear regression was calculated to predict delayed post-test scores based on 
duration of instruction and revisiting disposition (general propensity to revisit steps, both across and 
within days, M= 1.81, SD= 0.57). A significant regression equation was found (F(2, 661)= 7.82, pc.001) 
(Model 2, Table 5). Spending more time on the unit in total predicted higher scores on the delayed post¬ 
test, with an increase in the delayed post-test score as before. Propensity to revisit was not a significant 
predictor. Model 2 did not explain significantly more variance in delayed post-test scores than Model 1, 
r 2 = .022, p>.05. 

In Model 3, a multiple linear regression was calculated to predict delayed post-test scores based on 
duration of instruction and propensity to revisit specific steps (static illustration: M= 1.82, SD=1.80; 
dynamic visualization: M= 3.59, SD= 3.16), both across and within days. A significant regression equation 
was found (F(3, 660)= 5.89, pc.001) (Model 3, Table 5). Spending more time on the unit in total 
predicted higher scores on the delayed post-test, with an increase in the delayed post-test score as 
before. Each additional visit to the static illustration predicted a decrease in the delayed post-test score 
of 0.119. Additional visits to the dynamic visualization were not a significant predictor. Model 3 did not 
explain significantly more variance in delayed post-test scores than Model 1, r 2 = .026, p>.05. 

In Model 4, a multiple linear regression was calculated to predict delayed post-test scores based on the 
duration of instruction and distributing disposition (general propensity to revisit steps across days, 
M= 1.13, SD= 0.26). A significant regression equation was found (F(2, 661)= 7.34, pc.001) (Model 4, Table 
5). Spending more time on the unit in total predicted higher scores on the delayed post-test, with an 
increase in the delayed post-test score as before. Propensity to distribute study by revisiting steps across 
days was not a significant predictor. Model 4 did not explain significantly more variance in delayed post¬ 
test scores than Model 1, r 2 = .022, p>.05. 
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In Model 5, a multiple linear regression was calculated to predict delayed post-test scores based on 
duration of instruction and propensity to distribute study by revisiting specific steps (static illustration: 
M-X.21, SD= 0.66; dynamic visualization: M= 1.54, SD= 0.86) across days. A significant regression equation 
was found (F(3, 660)= 10.59, p<.001) (Model 5, Table 5). Spending more time on the unit in total 
predicted higher scores on the delayed post-test, with an increase in the delayed post-test score as 
before. Each additional visit to the static curriculum step predicted a decrease in the delayed post-test 
score of 0.534. Each additional visit to the dynamic curriculum step predicted an increase in the delayed 
post-test score of 0.430. Model 5 explained significantly more variance in the delayed post-test scores 
than Model 1, r 2 = .046, p<. 05. 


Table 5. Models of scores on the delayed post-test 


Standardized 

Unstandardized Coefficients 

Coefficients 



B 

Std. Error 

P t 

Model 1: Delayed post-test scores as a function of total time spent on the unit 

Intercept 

12.084 

0.238 

50.67** 

Total minutes for unit 

0.004 

0.001 

.139 3.60** 

Model 2: Delayed post-test scores 

as a function of total time spent on the unit and revisiting 



disposition 


Intercept 

12.224 

0.257 

47.61** 

Total minutes for unit 

0.004 

0.001 

.128 3.28** 

Average number of times 
each dyad visited steps 

across the unit 

-0.301 

0.186 

-.083 -1.62 

Model 3: Delayed post-test scores as a 

function of total time spent on the unit and revisiting specific 



steps 


Intercept 

12.07 

0.238 

50.69** 

Total minutes for unit 

0.005 

0.001 

.156 3.69** 

Total number of visits to 

static curriculum step 

-0.119 

0.055 

-.104 -2.15* 

Total number of visits to 

dynamic visualization step 

0.035 

0.032 

.054 1.11 

Model 4: Delayed post-test scores as a function of total time spent on the unit and distributing 



disposition 


Intercept 

12.420 

0.352 

35.24** 

Total minutes for unit 

0.005 

0.001 

.174 3.69** 
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Average number of days 

each dyad visited each -0.479 0.371 -.061 -1.29 

step 

Model 5: Delayed post-test scores as a function of total time spent on the unit and distributing visits 

to specific steps 


Intercept 

12.151 

0.251 


48.50* 

Total minutes for unit 

0.004 

0.001 

.130 

3.15*’ 

Number of days visited 
static curriculum step 

-0.534 

0.145 

-.172 

-3.67* 

Number of days visited 

0.430 

0.113 

.180 

3.80*’ 


dynamic visualization step 


Model 1 r 2 = .019, r 2 change**; Model 2 r 2 = .023, r 2 change NS; Model 3 (compared to Model 1) r 2 = 

.026, r 2 change NS; Model 4 (compared to Model 1) r 2 = .022, r 2 change NS; Model 5 r 2 = .046, r 2 
change**; * Significant at p<.05; ** Significant at pc.01 

7 DISCUSSION 

Our results are anchored to the realities of doing research in naturalistic classrooms. Our findings are 
correlational in nature, and reflect the messiness and complexities of doing research for and with 
complex science learning in public school settings. In such settings, teachers provide varied instructional 
supports outside the learning design, with some teachers allowing the learning design to do most of the 
work, others creating worksheets based on the learning design, and others creating entire labs to 
complement what students are learning. Teachers implement the learning design and associated 
measures according to schedules that they themselves may not have much control over as other testing 
and school events intercede. These challenges prevent strict experimental control, particularly for 
longer timescale interventions, yet it is important to study such interventions under real-world 
conditions. 

Using automatically collected log data still provides an excellent opportunity to pose theoretically driven 
research questions about learning, even under these conditions. However, results from such approaches 
are correlational in nature, and cannot rule out the possibility that some undetected variable might 
cause both the behaviour and the outcome. 

We took advantage of the log file data to test questions related to self-directed learning behaviours, 
namely, the amount of time a student spent studying the unit overall and various types of revisiting. In 
our study, the learning gains were modest, overall. Results across models show an advantage for longer 
time spent learning the unit, consistent with greater opportunity to learn the material. In our analysis, 
we found a small effect for duration of instruction. This is consistent with the value of spending more 
time on complex, inquiry activities documented in earlier studies of implementation of WISE units (Lee, 
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Linn, Varma, & Liu, 2010). However, this finding is correlational. It may be that students who spend 
longer do so because they know how to make the extra time benefit their learning. Simply requiring 
students to spend longer might not result in increased gains, as students who have the propensity to 
spend less time, might spend time reviewing material shallowly or without purpose, resulting in little 
benefit for learning. Thus, understanding more about ways students direct their restudy behaviour could 
be useful in supporting learning and guiding learning designs. 

Through a sequence of models, we tested various ways to operationalize revisiting as a metric: first, as a 
disposition to revisit in general; second, as a propensity to revisit specific material; third, as a disposition 
to distribute study over time; and fourth, as a propensity to distribute study of material over time. 
Revisiting previously studied materials is a compelling analytic because it is both theoretically grounded 
and relatively easy to detect. Prior research has shown that revisiting previously studied material — such 
as rereading a post in an online forum — can benefit learners (Wise, Hausknecht, & Zhao, 2014), and 
that repeated retrieval supports retention (Karpicke & Roediger, 2007). Likewise, repeated practice 
across days and over time supports learning (Andergassen et al., 2014; Modritscher et al., 2013). Our 
findings tell a more complicated story, suggesting that revisiting may not be as straightforward an 
analytic as one might hope. Past research has shown that students differ in their propensity to update 
their memories (Bjork, 1978), and can judge when it is advantageous to distribute their learning over 
time (Popham, 2009). Our sequence of models suggests that having a disposition to revisit previously 
studied material — whether with a lag between sessions or not — might not explain variance in 
retention. 

By testing a sequence of models that operationalized revisiting in different general and specific ways 
aligned to research on the value of distributed learning and a learning design, we were able to show 
that, as predicted, only distributed restudy of specific material supported retention. We found that 
while revisiting a dynamic visualization supported retention, revisiting a static illustration did not. In 
general, students visited the dynamic visualization with greater frequency and variability than they 
visited the static illustration, both in general and in a distributed (across days) manner. Overall, they 
visited the dynamic visualization nearly twice as often, suggesting a general perception that learning 
from the dynamic visualization required more visits. Alternatively, this could mean that students simply 
enjoyed playing with the dynamic visualization, and returned to it because of that. Students also spent 
more time per visit on the dynamic visualization (A4=134 seconds, 5D=119) than the static illustration 
(M =42 seconds, SD= 33). These findings are not surprising as the dynamic visualization takes time to use 
and interact with in order to access the information it contains. Further, the dynamic visualization 
contained additional information, relating energy transformation to global temperature, a detail not 
present in the static illustration. 

This detail is important as it highlights that this comparison is not intended to support inferences about 
the relative value of dynamic versus static visualizations in general. In fact, we stand by the decision to 
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include the static illustration, as our previous research has demonstrated that it supported students to 
connect their prior experiences and ideas and supported initial learning of a relatively abstract concept 
(Svihla & Linn, 2012a). In our sample, distributed restudy of the dynamic visualization positively 
benefited retention, whereas more frequent revisiting, in general or across days, of the static illustration 
predicted lower retention. We discuss this finding grounded in the Knowledge Integration framework. 
Revisiting the dynamic visualization is likely to elicit Knowledge Integration activities, such as 
distinguishing among alternative ideas and linking ideas. This is not simply because it is a dynamic, as 
opposed to static, visualization, but also because it contains more complex information that cannot be 
accessed without interacting with the visualization. Students had previously made observations with the 
dynamic visualization, using it to test their predictions and initial ideas. When revisiting, the dynamic 
visualization could cue memories of these, or students could conduct further tests, leading to new 
observations or new insights. These afford students the opportunity to evaluate their understanding and 
make new links between ideas across activities. The significant advantage of revisiting the dynamic 
visualization is consistent with evidence for their value when they are implemented with guidance in 
online units (McElhaney & Linn, 2011; Ryoo & Linn, 2012; Svihla & Linn, 2012a), and also consistent with 
a differentiation between simple recalling and more effortful relearning (Rawson & Dunlosky, 2011). 

However, it is important to note that pre-existing differences, such as in students' prior knowledge, their 
understanding of dynamic visualizations, as well as other factors may have led some students and not 
others to distribute their revisits across the different types of resources. Students who did not really 
develop understanding from their initial work with the interactive simulation might not have been 
predisposed to revisit it, relying instead on the easier to understand static visualization. Future research 
should investigate such variables under varied conditions to determine whether forced revisiting 
produces benefits for those who lack the propensity to revisit. 

The significant disadvantage for revisiting the static illustration is consistent with evidence that when 
students reread text, often adding multi-coloured underlining, they do not succeed as well as their peers 
who test themselves on the material or seek links among ideas in the instructional materials (Bjork & 
Bjork, 2009). These results resonate with other studies showing that durable, integrated understanding 
requires active integration of diverse ideas rather than recall of details (Bransford, Brown, & Cocking, 
2000; Linn & Eylon, 2011) and benefits from self-monitoring ability (White & Frederiksen, 1998). In 
addition, these results are consistent with research on spontaneous generation of explanations during 
learning. Based on our findings, more research is needed to understand how to design supports that 
encourage deliberate revisiting when learners are permitted to distribute their own practice. 

Findings also suggest that students who choose to revisit deceptively clear or less demanding 
information, such as the metaphor for energy transformation rather than more complex and difficult to 
interpret ideas in dynamic visualizations, do not monitor their own learning effectively. While we believe 
that the metaphor supported students to develop an initial understanding of energy transformation, we 
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also think that an overreliance on this simple explanation did not support students to develop coherent, 
integrated, durable understandings of climate change. Further research is needed to understand how to 
support students to focus their energies on restudying materials that create desirable difficulties; such 
activities may make learning more effortful, but they also make it more durable. 

Ultimately, the variables in Model 5 significantly predicted delayed post-test scores, but explained little 
variation in them. A number of other variables — e.g., pre- and post-assessments, student interest levels 
in the topic, other related instruction not documented by our system, differences in implementation — 
that we lacked measures for — might have better predicted delayed post-test scores. However, the 
objective of our modelling was not to account for maximum variance in delayed post-test scores, but 
rather to explore nuanced models related to our research questions. We argue that although distributed 
revisiting of specific steps explained little variance, it is still an interesting metric. The amount of time 
accounted for by students' revisits to these two steps is small — a matter of only one or two minutes. 
That those minutes can explain any variability in scores on a test taken days to weeks later is surprising 
yet theoretically backed by an extensive research base on distributed learning. 

7.1 Limitations 

This research was conducted in classrooms in only three schools and used a single inquiry unit, limiting 
generalizability. We focused on Knowledge Integration and the findings might not generalize to other 
outcome measures or learning designs. Our design was comparative, lacking an experimental control; as 
such, our findings are tentative. 

The relationships we found between revisiting and retention are correlational and could have been 
caused by some other unmeasured variable; for instance, there may be some systematic reason that 
lead certain students to revisit dynamic visualizations, and lead others to revisit static illustrations. For 
instance, teachers might have encouraged students they viewed as smarter to revisit the dynamic 
visualizations, and encouraged others to revisit the static illustrations. It is also possible that, like time 
on task, more adept students were better able to judge that revisiting the dynamic visualization would 
be helpful to their learning. Simply forcing students to revisit complex material does not mean they will 
know what to do with it. Future studies could investigate this through randomized, systematic 
comparisons, forcing some students to revisit material, and allowing others to direct their own 
revisiting. This would clarify whether the value of revisiting is in the revisit itself, or in the decision to 
revisit. Similarly, it would be helpful to better understand, through prompted recall or think-aloud 
protocol, more about how students make decisions to revisit specific materials. Alternatively, simple 
questions might be posed when a revisit to a particular step is detected, allowing students to reflect 
both on why they chose to revisit and whether they found it beneficial to do so. This could lead to new 
metrics that might differentiate between reasons for revisiting. This would provide guidance for the 
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creation of learning designs that aim to encourage students to make the decision to revisit specific 
material, rather than forcing them down a particular path. 

We defined revisiting based on activities detected in log files. Students could engage in other forms of 
revisiting that our methods do not document. For instance, teachers in some classes may have retaught 
information, or provided study sheets and additional assessments covering related content. 

Although we found variables that could significantly predict the variability of delayed post-test scores, 
other explanations should be entertained. For instance, the finding that spending longer on the unit 
predicted higher scores could be a reflection of some other unmeasured variable, such as differences in 
student persistence, or in teachers' expectations of individual students. Thus, although this finding 
mirrors previous research, it should not be viewed as a prescriptive that more time is necessarily better. 
Likewise, there may be other explanations for why students revisit a particular step; they may be 
prompted by a teacher based on a conversation not shared across the class, meaning that the 
conversation itself may have produced the benefit, rather than the opportunity to restudy. 

7.2 Implications 

Combining learning analytics with learning design has previously helped guide the analysis of and 
refinement of learning designs in learning management systems (Wise, 2014; Wise, Saghafian, & 
Padmanabhan, 2012). This combination allows researchers to plan and test learning designs guided by 
theories of learning (Lockyer et a I., 2013; Wise, 2014). 

Our sequence of models focused on retention, as opposed to initial learning, and this has implications 
for learning designers who wish to incorporate revisiting as a learning analytic. First, based on research 
on distributed learning, it is likely that the benefit of revisiting may not be visible on an immediate post¬ 
test (Rohrer & Taylor, 2006). This means that detecting benefits on retention requires longer-term 
investments in the study sites. This can present challenges, especially for researchers working in schools 
already beleaguered by the amount of testing. Such settings may prevent researchers from 
implementing a delayed post-test, and therefore would present a barrier to furthering this work. 

We found that revisiting is a promising yet complex learning analytic for predicting retention of coherent 
understanding of complex scientific phenomena. Its value as an analytic was increased by theoretical 
guidance for what is revisited. While revisiting as a broad, generic metric did not predict retention, a 
more nuanced approach to revisiting did. This metric aligns with the Knowledge Integration framework, 
in that revisiting dynamic visualizations better supports the kind of effortful relearning needed for 
coherent understanding and retention of complex scientific phenomena. 

The greater variability in visits to the dynamic visualization, compared to the static illustration, suggests 
implications for instructors and for learning designers working with analytics. Not all students 
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recognized the need to restudy the complex information in the dynamic visualization. Although 
tentative, our findings suggest that there may be a benefit to prompting students to restudy such 
material, either by the instructor or by the learning management system. Further research is needed to 
contrast the relative benefits from spontaneous and prompted restudy, however, as prompted restudy 
may not have the same effect. 

Revisiting is a valuable and relatively easy to detect analytic for retention that could be applied to other 
contexts; however, it should be applied in a nuanced manner, informed by the theory of learning guiding 
the learning design. 

The finding that generic revisiting — even distributed across days — did not significantly predict 
retention suggests two implications for learning designers using learning analytics. The first is promising, 
in that it suggests that a general disposition to revisit, something learning designers have little control 
over, might not be that important as a concern. However, without an additional measure of revisiting as 
a disposition, this should be treated as tentative and explored further in other settings. The second 
implication is that a nuanced design and definition of revisiting is likely needed and likely contextual. 
Generic restudy of all material — a common instructional approach we observed in classrooms — may 
not be as beneficial for retention as one would hope. Yet, this suggests that there is much more work to 
be done to understand how to apply revisiting as an analytic in specific learning designs, and this study 
contributes and contrasts a range of ways to do so. 

Given the relatively brief time required to prompt and carry out distributed restudy, either as an 
instructor or as a learning designer, our findings suggest it may be a worthwhile investment. However, 
one of the lessons from our sequence of models is that not all revisiting is equally capable of predicting 
retention. Understanding how to incorporate findings from the research on distributed learning into a 
specific learning design, and further how to use revisiting as an analytic appears to require a nuanced 
understanding of both the theory and the learning design. Based on our findings, we encourage 
instructors and learning designers to focus the efforts of distributed restudy on the most central and 
complex material in their learning designs. 

The potential of revisiting as an analytic, at least as we have conceptualized and studied it, is most 
promising for learning theories that address long-term and longitudinal approaches to learning. We see 
potential in using such analytics to guide the learning designs that, for instance, support learning 
progressions (Duncan & Hmelo-Silver, 2009; Gunckel, Mohan, Covitt, & Anderson, 2012; Shin & Stevens, 
2012) and curricular standards based (at least in part) on them (e.g., NGSS Lead States, 2013; National 
Governors Association Center for Best Practices, & Council of Chief State School Officers, 2010). In such 
contexts, revisiting as an analytic could help refine learning theory and learning designs by helping 
researchers maintain focus on both initial learning supports and longer-term retention. 
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Adding analytic tools to learning designs like WISE could also help teachers pose, then monitor, long¬ 
term retention questions. Many of our teachers were interested in posing their own questions about the 
impact their instruction had when using WISE. They were curious, for instance, if a whole-class demo of 
an interactive visualization, after students had encountered it in pairs, could improve student retention. 
As many of them taught multiple sections, they sometimes ran informal comparisons, trying one 
approach with one section and another with another section. For instance, one teacher talked about 
using a simple starter prompt with one section, and revisiting a specific question from the learning 
design with another section. Integrated analytics tools could capitalize on this interest and help them 
structure meaningful investigations, including providing guidance to look beyond short-term gains. Such 
a suite of tools could include options to help teachers select central concepts they planned to guide 
students to revisit as well as those they observed students spontaneously revisiting, a set of prompts to 
encourage teachers to record notes about their instruction in room as a way to gather information not 
automatically recorded, and automatically computed scores for revisiting. Such tools could elevate 
teaching practice by supporting teachers to pursue more easily what might otherwise be fleeting 
curiosities. 
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